Redefining the User Interface: Augmented Reality

Virtual worlds (a.k.a. virtual realities) are 3D computer-generated environments created through the use of high-performance computers, 3D interaction devices, and 3D display systems. Most of the virtual worlds systems that have been so well publicized over the past five years use head-mounted displays that block off the wearer from the surrounding real world, effectively immersing her within a synthesized world. These systems can be particularly effective for many applications, ranging from fantasy games to scientific research. In contrast, we believe that the most powerful and commonplace virtual worlds of the future will not replace the real world, but will rather augment it with additional information. This approach is called augmented reality and was pioneered by Ivan Sutherland, who, over a quarter century ago, developed the first see-through head-mounted display [8]. His system presented graphics to the user on a pair of stereo displays, worn on the user’s head. The image produced by the displays was combined with the user’s view of the world using half-silvered mirrors. A 3D tracking system determined the position and orientation of the user’s head. This enabled the system to change the view, based on the direction in which the wearer was looking. Consider some of the possibilities: Graphics, text, and sound overlaid on the surrounding world could explain how to operate, maintain, or repair equipment, without requiring that the user refer to a separate paper or electronic manual. Similarly, participants in a business meeting could interact with a dynamic shared financial or organizational model, represented in 3D and selectively supplemented with each user’s private personal annotations. This brief paper provides an overview of the work being done at Columbia’s Computer Graphics and User Interfaces Laboratory to address fundamental issues in designing effective augmented reality systems. We describe “hybrid user interfaces” that merge 2D and 3D displays and interaction devices, augmented realities that annotate the surrounding physical world with knowledge-based 3D graphics, and augmented realities that extend a contemporary window-based environment so that it becomes part of a wearable information surround. These projects all use the head-mounted display shown in Figure 1. We assembled it from a Reflection Technology Private Eye, a Logitech ultrasonic 3D position and orientation tracker, and a mirror beam splitter. The Private Eye is a small 720 × 280 resolution bitmapped display [6]. We positioned the beam splitter so that the user can see the surrounding world through it and the Private Eye reflected from it. The triangle at the top of the visor contains the sensors for the ultrasonic tracker.